Systems and methods for training a neural network for estimating a trajectory of a vehicle

ABSTRACT

Systems and methods for training a neural network for estimating a trajectory of a vehicle are disclosed. In one embodiment, a system includes one or more processors, and a non-transitory computer-readable medium storing computer-readable instructions. The computer-readable instructions cause the one or more processors to receive sensor data of a plurality of examples from a plurality of vehicle sensors, input the sensor data into a sensor data neural network to generate a sensor data intermediate space and receive structured data of the plurality of examples. The computer-readable instructions cause the one or more processors to input the structured data into a structured data neural network to generate a structured data intermediate space, calculate a first loss between the sensor data intermediate space and the structured data intermediate space using a first loss function, and provide the first loss to the sensor data neural network and the structured data neural network.

TECHNICAL FIELD

The present specification relates to training neural networks, and moreparticularly, to training a neural network for estimating a trajectoryof a vehicle using modified structured data.

BACKGROUND

Autonomous and semi-autonomous vehicles may utilize a neural network toestimate vehicle trajectories based on sensor input regarding theenvironment. Commonly, the neural network is trained using input sensordata, such as images from cameras, point cloud data from a LiDAR sensor,and the like. Thus, training the neural network requires actual sensordata, which may be limited and not account for all possible drivingscenarios. For example, no sensor data may exist for a particulardriving scenario and thus the neural network may not accurately estimatea trajectory of the vehicle in such a scenario.

SUMMARY

In one aspect, a system for training a neural network for estimating atrajectory of a vehicle includes one or more processors and anon-transitory computer-readable medium storing computer-readableinstructions. The computer-readable instructions, when executed by theone or more processors, cause the one or more processors to receivesensor data of a plurality of examples from a plurality of vehiclesensors, input the sensor data into a sensor data neural network togenerate a sensor data intermediate space and receive structured data ofthe plurality of examples. The computer-readable instructions furthercause the one or more processors to input the structured data into astructured data neural network to generate a structured dataintermediate space, calculate a first loss between the sensor dataintermediate space and the structured data intermediate space using afirst loss function, and provide the first loss to the sensor dataneural network and the structured data neural network.

In another aspect, a method for training a neural network for estimatinga trajectory of a vehicle includes receiving sensor data of a pluralityof examples from a plurality of vehicle sensors, and inputting thesensor data into a sensor data neural network to generate a sensor dataintermediate space. The method further includes receiving structureddata of the plurality of examples, inputting the structured data into astructured data neural network to generate a structured dataintermediate space, calculating a first loss between the sensor dataintermediate space and the structured data intermediate space using afirst loss function, and providing the first loss to the sensor dataneural network and the structured data neural network

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments set forth in the drawings are illustrative and exemplaryin nature and not intended to limit the disclosure. The followingdetailed description of the illustrative embodiments can be understoodwhen read in conjunction with the following drawings, where likestructure is indicated with like reference numerals and in which:

FIG. 1 schematically depicts an example environment of a vehicle;

FIG. 2 schematically depicts an example vehicle, according to one ormore embodiments described and illustrated herein;

FIG. 3 schematically depicts an example dataflow diagram of a method fortraining a neural network for estimating a trajectory of a vehicleaccording to one or more embodiments described and illustrated herein;and

FIG. 4 schematically depicts an example system for training a neuralnetwork for estimating a trajectory of a vehicle according to one ormore embodiments described and illustrated herein.

DETAILED DESCRIPTION

The embodiments disclosed herein are directed to vehicles, systems andmethods for training a neural network for estimating a trajectory of avehicle. Training a neural network for an autonomous or semi-autonomousvehicle involves inputting sensor data and known driver behavior so thatthe neural network can learn how to navigate many different drivingscenarios. As an example, one scenario is how to brake the vehicle whenthere is a red traffic light and there is a vehicle slowing down aheadof the ego vehicle. As used here, the term “ego vehicle” means thevehicle having the neural network and generating sensor data from aplurality of sensors used to both train the neural network and operatein the environment.

To train the neural network for a particular driving scenario, theneural network receives sensor data corresponding to that particulardriving scenario or a similar driving scenario. Sensor data may include,but is not limited to, video data, LiDAR data, radar data,time-of-flight sensor data, and the like. However, generating sensordata from real-life vehicles is time consuming and costly. Additionally,real-life sensor data may cover only a very small fraction of thenear-infinite number of driving scenarios that a driver or an autonomousvehicle may encounter.

Simulation data may be used to train the neural network in addition to,or in lieu of, real-life sensor data from physical sensors. Simulationdata simulates the sensor data itself, which is then inputted into theneural network for training purposes. As an example, a video simulationmay simulate a particular driving scenario from an ego vehicle'sperspective. Such a video simulation may be computer-generated, forexample. However, generating video simulations may also be timeconsuming, costly, and require a significant amount of computerprocessing power.

Thus, training data for particular scenarios in the form of real-lifesensor data or simulation data may not be available for many drivingscenarios.

Embodiments of the present disclosure improve on training neuralnetworks by enabling structured data to be easily manipulated in amanner such that unique driving scenarios may be generated and inputtedinto the neural network easily and quickly. As used herein “structureddata” means data other than raw sensor data, and includes, but is notlimited to, outputs of a detection algorithm (e.g., lane detection,vehicles and other road agents detected by an object recognitionalgorithm, etc.), map data (e.g., high definition map data), or anyother observed or generated data that represents the environment of avehicle. “Sensor data” is the raw data that is generated by one or morephysical sensors on a vehicle in the environment.

As described in more detail below, a method of training a neural networkto predict vehicle trajectories comprises two neural network paths: asensor data path and structured data path. In the sensor data path, rawsensor data (e.g., image data, LiDAR data, and the like) is inputtedinto a sensor data neural network that outputs a sensor dataintermediate space (i.e., latent space), which may then be inputted intoa sensor data reconstruction neural network that recreates the inputsensor data. A loss function may be applied to calculate the lossbetween the input sensor data and the reconstructed sensor data tominimize the loss therebetween. Similarly, in the structured data path,structured data (e.g., detected lanes, detected obstacles, map data, andthe like) corresponding to the environment generating the sensor data isinputted into a structured data neural network that outputs a structureddata intermediate space (i.e., latent space), which may then be inputtedinto a structured data reconstruction neural network that recreates thestructured data. A loss function may be applied to calculate the lossbetween the input structured data and the reconstructed structured datato minimize the loss therebetween.

A loss between the sensor data intermediate space and the structureddata intermediate space is determine by a loss function. This loss isthen fed back to the sensor data neural network and the sensor dataneural network to decrease the loss therebetween over time. Thus, thesensor data intermediate space and the structured data intermediatespace may become aligned over time after a plurality of examples areinputted into the sensor data neural network and the structured dataneural network.

Data from the structured data intermediate space or the sensor dataintermediate space may be inputted into a trajectory planning neuralnetwork to estimate one or more vehicle trajectories. Aligning thestructured data intermediate space and the sensor data intermediatespace may create a more robust system in that both sensor data andstructured data are used to train the overall neural network andtherefore make trajectory predictions.

Use of the structured data intermediate space as input into thetrajectory planning neural network enables the easy generation of verydiverse traffic scenarios. Rather than generating full sensor data(e.g., video data or point cloud data) or simulated sensor data,modified structured data may be generated by merely moving points and orlines in the structured data. For example, a non-existent car may beadded to structured data based on previous sensor data by changing oneor more variables in the structured data. Such a process is much simplerthan adding a vehicle to video simulation data or a simulated pointcloud. Thus, the modified structured data may be used to generate a vastnumber of different driving scenarios for estimating vehicletrajectories.

Various embodiments of systems and methods for training a neural networkfor estimating vehicle trajectories are described below.

Referring now to FIG. 1 , a non-limiting example of a vehicleenvironment 10 is schematically depicted. The environment 10 has a firstroad 12 with a first lane 13A for travel in one direction and a secondlane 13B for travel in an opposite direction. The environment 10 furtherincludes an intersection 19 wherein a second road 14 intersects with thefirst road 12. The second road has a first lane 15A for travel in afirst direction and a second lane 15B for travel in a second direction.An ego vehicle (not shown) operating in the environment 10 of FIG. 1 maybe some distance behind a vehicle 20 in the second lane 13B as both thevehicle 20 and the ego vehicle approach a stop sign 16. Other objectsmay be in the environment, such as a tree 18 and a bicyclist 17.

Referring now to FIG. 2 , a non-limiting example of a vehicle 130 (e.g.,an ego vehicle) having sensors 208 is schematically illustrated. Thevehicle 130 may be driven by a human driver and collect sensor datausing the sensors 208 for various driving scenarios. This sensor datamay then be used to train the neural networks as described in moredetail below.

The sensors 208 may be any known or yet-to-be-developed sensors.Non-limiting example sensors include video cameras, LiDAR sensors, radarsensors, time-of-flight sensors, proximity sensors, and the like. Thesensors 208 capture data of the environment. Referring again to FIG. 1 ,sensors 208 configured as video cameras capture video of the vehicle 20,the stop sign 16, the intersection 19, and the like. In the case ofLiDAR sensors, point cloud data of the various objects in theenvironment 10 is generated.

In addition, structured data of the environment 10 is gathered and/orgenerated from the sensor data. In the case of gathered structured data,high definition map data of the environment 10 may be accessed, such asfrom a remote database. The high definition map data may includeinformation such as the number of lanes, the geometry of the lanes, thespeed limit, the geometry of an intersection, buildings and otherobjects in the area, and the like. In the case of generating structureddata, one or more object recognition algorithms may be executed on thesensor data generated by the one or more sensors to detect the lanes andobjects within the environment. The one or more object recognitionalgorithms may be executed on the vehicle 130 and/or offline at a remoteserver, for example. Any known or yet-to-be-developed object recognitionalgorithms may be used to detect the objects in the environment. Forexample, referring to FIG. 1 , an object recognition algorithm maydetect vehicle 20, and generate a bounding box around it that isrepresented in the structured data. The existence of the vehicle 20 andattributes and relationship to the ego vehicle may be stored in thestructured data. It should be understood that embodiments of the presentdisclosure are not limited by any format for the structured data, andthat the structured data may take on any format and/or organization.

Referring now to FIG. 3 , a non-limiting method of a dataflow diagram100 for training a neural network for estimating a vehicle trajectory isillustrated. The illustrated dataflow comprises a sensor data path 110and a structured data path 150. The sensor data path 110 will bedescribed followed by a description of the structured data path 150.Sensor data 112 may represent a plurality of vehicle maneuver examples.As stated above, the sensor data 112 represents the environment asobserved by the sensors of the ego vehicle.

The sensor data 112 is provided to sensor data neural network 114. Thesensor data neural network 114 creates a sensor data intermediate space116, which may be a low dimensional a latent space, for example.Embodiments are not limited by the type of neural network employed bythe sensor data neural network 114. Any known or yet-to-be-developedneural network may be used to generate the sensor data intermediatespace 116. As a non-limiting example, the sensor data intermediate space116 may be a vector comprising about 100 latent variables.

Next, the sensor data intermediate space 116 is provided as input to asensor data reconstruction neural network 118. The sensor datareconstruction neural network 118 uses the sensor data intermediatespace 116 to reconstruct the sensor data 112 that was provided as inputto the sensor data neural network 114. However, differences may existbetween the sensor data 112 and the reconstructed sensor data 120 thatis outputted from the sensor data reconstruction neural network 118.Although not shown, a loss function may calculate a difference (i.e.,loss) between the sensor data 112 and the reconstructed sensor data 120.The output of the loss function may be provided to the sensor dataneural network 114 and/or the sensor data reconstruction neural network118 to minimize the loss calculated by the loss function over time. Insome embodiments, the sensor data reconstruction neural network 118 isnot used.

The structured data path 150 may operate in parallel with the sensordata path 110. Structured data 152 corresponding to the environmentcaptured by the sensor data 112 is provided as input to a structureddata neural network 154. The structured data may be observed orgenerated structured data as described above. The structured data neuralnetwork 154 creates a structured data intermediate space 156, which maybe a low dimensional latent space, for example. Embodiments are notlimited by the type of neural network employed by the structured dataneural network 154. Any known or yet-to-be-developed neural network maybe used to generate the structured data intermediate space 156. As anon-limiting example, the structured data intermediate space 156 may bea vector comprising about 100 latent variables.

The structured data intermediate space 156 may be provided as input to astructured data reconstruction neural network 158. The structured datareconstruction neural network 158 uses the structured data intermediatespace 156 to reconstruct the structured data 152 that was provided asinput to the structured data neural network 154. However, differencesmay exist between the structured data 152 and the reconstructedstructured data 160 that is outputted from the structured datareconstruction neural network 158. Although not shown, a loss functionmay calculate a difference (i.e., loss) between the structured data 152and the reconstructed structured data 160. The output of the lossfunction may be provided to the structured data neural network 154and/or the structured data reconstruction neural network 158 to minimizethe loss calculated by the loss function. In some embodiments, thestructured data reconstruction neural network 158 is not used.

The sensor data path 110 and the structured data path 150 are linked bya loss function 170 that receives as inputs the sensor data intermediatespace 116, which may be all of the data of the sensor data intermediatespace 116 or some sub-set thereof, and the structured data intermediatespace 156, which may be all of the data of the structured dataintermediate space 156 or some sub-set thereof. The loss function 170calculates a loss 172 (i.e., a difference) between the sensor dataintermediate space 116 and the structured data intermediate space 156.The loss 172 is provided to the sensor data neural network 114 and orthe structured data neural network 154 to decrease subsequentlycalculated losses between the two intermediate spaces over time. In thismanner, the sensor data influences the structured data intermediatespace 156 and vice-versa, thereby providing a more robust neural networkwith simply sensor data or structured data alone.

A plurality of examples in the form of sensor data 112 and structureddata 152 are inputted into the respective neural networks until thevarious loss functions are minimized, for example.

Still referring to FIG. 3 , the structured data intermediate space 156is provided as an input to a trajectory estimation neural network 182that outputs one or more estimated vehicle trajectories 184 based on thestructured data intermediate space 156. The one or more estimatedvehicle trajectories 184 represent ideal trajectories for an ego vehiclebased on the environment as represented by the sensor data 112 and/orthe structured data 152. In some embodiments, another loss function 188receives as input the one or more estimated vehicle trajectories 184 andone or more ground truth trajectories 186, and calculates a loss 190that is then provided to the structured data neural network 154 tominimize the loss 190 over time. The one or more ground truthtrajectories 186 may take on any form. As one non-limiting example, theone or more ground truth trajectories may be an actual trajectory takenby the driver of the ego vehicle. Thus, the loss function may compare anestimated trajectory with an actual trajectory taken by the driver overa period of time.

Embodiments of the present disclosure enable easy creation of many, manydifferent driving scenarios to train a neural network, which in thepresent example is the structured data neural network 154. To generatedifferent driving scenarios, a pre-existing structured data example(i.e., observed structured data) may be manipulated to change theenvironment represented by the pre-existing structured data. Forexample, lanes may be shifted, added or removed, intersections may bechanged, objects such as vehicles may be shifted, added or removed, andthe like. The representations of objects in the structured data aresimple and easily modified. For example, to add a vehicle to thestructure data, only a few variable may be changed. This is in contrastto generating simulated sensor data, which requires recreation of thescene with simulation video or a simulated point cloud, for example.Even further, the use of modified structured data enables the structureddata neural network 154 and the trajectory estimation neural network tobe trained without the need for continued collection of sensor data andstructured data of actual environments.

Many, many modified structured data examples may be used to simulatedriving scenarios where perhaps there is a lack of data.

Referring now to FIG. 4 , a non-limiting, example system 200 fortraining a neural network for estimating vehicle trajectories isillustrated. The system 200 includes one or more processors 206, acommunication path 204, one or more memory modules 202 configured asnon-transitory computer-readable medium, a wireless communication module212 (e.g., network interface hardware for communication with externalcomputing devices, vehicles, and the like), one or more sensors 208, andone or more additional memory modules 210, the details of which will beset forth in the following paragraphs. It should be understood that thesystem 200 of FIG. 4 is provided for illustrative purposes only, andthat other systems comprising more, fewer, or different components maybe utilized.

Each of the one or more processors 206 may be any device capable ofexecuting machine readable and executable instructions. Accordingly,each of the one or more processors 206 may be a controller, anintegrated circuit, a microchip, a computer, or any other computingdevice. The one or more processors 206 are coupled to a communicationpath 204 that provides signal interconnectivity between variouscomponents of the system 200. Accordingly, the communication path 204may communicatively couple any number of processors 206 with oneanother, and allow the components coupled to the communication path 204to operate in a distributed computing environment. Specifically, each ofthe components may operate as a node that may send and/or receive data.As used herein, the term “communicatively coupled” means that coupledcomponents are capable of exchanging data signals with one another suchas, for example, electrical signals via conductive medium,electromagnetic signals via air, optical signals via optical waveguides,and the like.

Accordingly, the communication path 204 may be formed from any mediumthat is capable of transmitting a signal such as, for example,conductive wires, conductive traces, optical waveguides, or the like. Insome embodiments, the communication path 204 may facilitate thetransmission of wireless signals, such as WiFi, Bluetooth®, Near FieldCommunication (NFC) and the like. Moreover, the communication path 204may be formed from a combination of mediums capable of transmittingsignals. In one embodiment, the communication path 204 comprises acombination of conductive traces, conductive wires, connectors, andbuses that cooperate to permit the transmission of electrical datasignals to components such as processors, memories, sensors, inputdevices, output devices, and communication devices. Accordingly, thecommunication path 104 may comprise a vehicle bus, such as for example aLIN bus, a CAN bus, a VAN bus, and the like. Additionally, it is notedthat the term “signal” means a waveform (e.g., electrical, optical,magnetic, mechanical or electromagnetic), such as DC, AC,sinusoidal-wave, triangular-wave, square-wave, vibration, and the like,capable of traveling through a medium.

The system 200 includes one or more memory modules 202 coupled to thecommunication path 204. The one or more memory modules 202 may compriseRAM, ROM, flash memories, hard drives, or any device capable of storingmachine readable and executable instructions 203 such that the machinereadable and executable instructions can be accessed by the one or moreprocessors 206 and execute the functionalities described herein. Themachine readable and executable instructions may comprise logic oralgorithm(s) written in any programming language of any generation(e.g., 1GL, 2GL, 3GL, 4GL, or 5GL) such as, for example, machinelanguage that may be directly executed by the processor, or assemblylanguage, object-oriented programming (OOP), scripting languages,microcode, etc., that may be compiled or assembled into machine readableand executable instructions and stored on the one or more memory modules202. Alternatively, the machine readable and executable instructions maybe written in a hardware description language (HDL), such as logicimplemented via either a field-programmable gate array (FPGA)configuration or an application-specific integrated circuit (ASIC), ortheir equivalents. Accordingly, the methods described herein may beimplemented in any conventional computer programming language, aspre-programmed hardware elements, or as a combination of hardware andsoftware components.

Referring still to FIG. 4 , the system 200 may optionally include one ormore additional memory modules 210 that store data relevant toperforming the functionalities described herein. For example the one ormore additional memory modules 210 may include map data, historicalsensor data, other structured data, and other information used to trainthe neural networks described herein. This additional information may bestored in the one or more memory modules 202 or in additional memorymodules 210 as shown in FIG. 4 . In some embodiments, all or some of theinformation stored in the one or more additional memory modules may bestored remotely in a server device. Other variations for storing thisadditional information are also possible.

The system 200 comprises one or more sensors 208 deployed on vehicles130 (see FIG. 2 ). The one or more sensors 208 may include, but are notlimited to, LiDAR sensors, RADAR sensors, optical sensors (e.g.,cameras, laser sensors, proximity sensors, location sensors), and thelike. The sensors 208 produce sensor data that may be used to perform avariety of functions, such as train a neural network, as described indetail above.

Still referring to FIG. 4 , the example system 200 comprises wirelesscommunication module 212 (e.g., network interface hardware) forcommunicatively coupling the system 200 to remote computing devices, aremote server, and the like. The wireless communication module 212 canbe communicatively coupled to the communication path 204 and can be anydevice capable of transmitting and/or receiving data via a network.Accordingly, the wireless communication module 212 can include acommunication transceiver for sending and/or receiving any wired orwireless communication. For example, the wireless communication module212 may include an antenna, a modem, Wi-Fi card, WiMax card, mobilecommunications hardware, near-field communication hardware, satellitecommunication hardware and/or any wireless hardware for communicatingwith other networks and/or devices. In one embodiment, the wirelesscommunication module 212 includes hardware configured to operate inaccordance with the Bluetooth® wireless communication protocol.

It should now be understood that embodiments of the present disclosureare directed to systems and methods for training a neural network forestimating a vehicle trajectory. Embodiments enable the easy developmentof a plethora of driving scenarios by simply modifying structured datawithout the need for generating simulation data, such as video data. Thesystem combines both sensor data and structured data to initially trainthe neural network. Then, modified structured data may be used tocontinue to train the neural network using many, many different drivingscenarios without collected sensor data and collected structured data,thereby more precisely training the neural network without the time andcost of generating actual sensor data and/or simulated sensor data.

It is noted that the terms “substantially” and “about” may be utilizedherein to represent the inherent degree of uncertainty that may beattributed to any quantitative comparison, value, measurement, or otherrepresentation. These terms are also utilized herein to represent thedegree by which a quantitative representation may vary from a statedreference without resulting in a change in the basic function of thesubject matter at issue.

While particular embodiments have been illustrated and described herein,it should be understood that various other changes and modifications maybe made without departing from the spirit and scope of the claimedsubject matter. Moreover, although various aspects of the claimedsubject matter have been described herein, such aspects need not beutilized in combination. It is therefore intended that the appendedclaims cover all such changes and modifications that are within thescope of the claimed subject matter.

The invention claimed is:
 1. A system for training a neural network forestimating a trajectory of a vehicle, the system comprising: one or moreprocessors; and a non-transitory computer-readable medium storingcomputer-readable instructions that, when executed by the one or moreprocessors, cause the one or more processors to: receive sensor data ofa plurality of vehicle maneuver examples from a plurality of vehiclesensors; input the sensor data into a sensor data neural network togenerate a sensor data intermediate space; receive structured data ofthe plurality of vehicle maneuver examples; input the structured datainto a structured data neural network to generate a structured dataintermediate space; calculate a first loss between the sensor dataintermediate space and the structured data intermediate space using afirst loss function; provide the first loss to the sensor data neuralnetwork and the structured data neural network; input the structureddata intermediate space data into a trajectory estimation neural networkto generate one or more vehicle trajectories; and control the vehicleaccording to the one or more vehicle trajectories.
 2. The system ofclaim 1, wherein the first loss function minimizes the first loss. 3.The system of claim 1, wherein the computer-readable instructionsfurther cause the one or more processors to: input sensor dataintermediate space data from the sensor data intermediate space into asensor data reconstruction neural network to generate reconstructedsensor data; calculate a second loss between the received sensor dataand the reconstructed sensor data using a second loss function; andinput the second loss into at least one of the sensor data neuralnetwork and the sensor data reconstruction neural network.
 4. The systemof claim 1, wherein the computer-readable instructions further cause theone or more processors to: input structured data intermediate space datafrom the structured data intermediate space into a structure datareconstruction neural network to generate reconstructed structured data;calculate a third loss between the received structured data and thereconstructed structured data using a third loss function; and input thethird loss into at least one of the structured data neural network andthe structured data reconstruction neural network.
 5. The system ofclaim 1, wherein the computer-readable instructions further cause theone or more processors to: calculate a fourth loss between the one ormore vehicle trajectories and one or more ground truth trajectoriesusing a fourth loss function; and input the fourth loss into thestructured data neural network.
 6. The system of claim 1, wherein thecomputer-readable instructions further cause the one or more processorsto: receive manipulated structured data; and input the manipulatedstructured data into the structured data neural network such that theone or more vehicle trajectories are based on the manipulated structureddata.
 7. The system of claim 1, wherein the structured data comprises atleast one of a position of a lane and a vehicle.
 8. The system of claim1, wherein the sensor data is image data.
 9. A method for training aneural network for estimating a trajectory of a vehicle, the methodcomprising: receiving sensor data of a plurality of vehicle maneuverexamples from a plurality of vehicle sensors; inputting the sensor datainto a sensor data neural network to generate a sensor data intermediatespace; receiving structured data of the plurality of vehicle maneuverexamples; inputting the structured data into a structured data neuralnetwork to generate a structured data intermediate space; calculating afirst loss between the sensor data intermediate space and the structureddata intermediate space using a first loss function; and providing thefirst loss to the sensor data neural network and the structured dataneural network; inputting the structured data intermediate space datainto a trajectory estimation neural network to generate one or morevehicle trajectories; and controlling the vehicle according to the oneor more vehicle trajectories.
 10. The method of claim 9, wherein thefirst loss function minimizes the first loss.
 11. The method of claim 9,further comprising: inputting sensor data intermediate space data fromthe sensor data intermediate space into a sensor data reconstructionneural network to generate reconstructed sensor data; calculating asecond loss between the received sensor data and the reconstructedsensor data using a second loss function; and inputting the second lossinto at least one of the sensor data neural network and the sensor datareconstruction neural network.
 12. The method of claim 9, furthercomprising: inputting structured data intermediate space data from thestructured data intermediate space into a structure data reconstructionneural network to generate reconstructed structured data; calculating athird loss between the received structured data and the reconstructedstructured data using a third loss function; and inputting the thirdloss into at least one of the structured data neural network and thestructured data reconstruction neural network.
 13. The method of claim9, further comprising: calculating a fourth loss between the one or morevehicle trajectories and one or more ground truth trajectories using afourth loss function; and inputting the fourth loss into the structureddata neural network.
 14. The method of claim 9, further comprisingreceiving manipulated structured data; and inputting the manipulatedstructured data into the structured data neural network such that theone or more vehicle trajectories are based on the manipulated structureddata.
 15. The method of claim 9, wherein the structured data comprisesat least one of a position of a lane and a vehicle.
 16. The method ofclaim 9, wherein the sensor data is image data.